A ResNet-Based Audio-Visual Fusion Model for Piano Skill Evaluation

With the rise in piano teaching recent years, many people have joined ranks of learners. However, high cost traditional manual instruction and exclusive one-on-one model made learning an extravagant endeavor. Most existing approaches, based on audio modality, aim to evaluate players’ skills. Unfortunately, these methods overlook information contained videos, resulting a one-sided simplistic evaluation player’s More recently, multimodal-based been proposed assess skill level players by using both video information. multimodal approaches use shallow networks extract features, which limits their ability complex spatio-temporal time-frequency characteristics from performances. Furthermore, fingering pitch-rhythm performance is embedded within respectively. Therefore, we propose ResNet-based audio-visual fusion that able visual features finger movement track auditory including pitch rhythm. The joint are then obtained through feature technique capturing correlation complementary between audio, enabling comprehensive accurate level. Moreover, can temporal frequency Firstly, ResNet18-3D used as backbone network for our branch, allowing us data. Then, utilize ResNet18-2D aural branch extracted fused with generating final evaluation. experimental results PISA dataset show model, validation accuracy 70.80% average training time 74.02 s, outperforms baseline terms operational efficiency. explore impact different layers ResNet model’s performance. In general, achieves optimal when ratio balanced. best achieved 68.70% differs significantly.